learning 
and skills 
development 
agency 



research report 



delivering quality and choice 

how performance indicators help 
and how performance indicators hinder 

Adrian Perry 
and Matthew Simpson 




learning 
and skills 
development 
agency 



research report 



delivering quality and choice 

how performance indicators help 
and how performance indicators hinder 

Adrian Perry 
and Matthew Simpson 



Headmaster: ‘Shall I tell you what is wrong with Hector as 
a teacher? It isn’t that he doesn’t produce results. He does. 

But they are unpredictable and unquantifiable and in the 
current educational climate, that is no use. He may well 
be doing his job, but there is no method that I know of 
that enables me to assess the job that he is doing.’ 

The History Boys’, Alan Bennett 



Published by the Learning and Skills Development Agency 
www.lsda.org.uk 

LSDA is committed to providing publications that are accessible to all. To request 
additional copies of this publication ora different format, please contact: 

Information Services 

Learning and Skills Development Agency 

Regent Arcade House 

19-25 Argyll Street 

London W1F 7LS 

Tel 020 7297 9144 

Fax 020 7297 9242 

enquiries@LSDA.org.uk 

From 1 April details available from the website for the Learning and Skills 
Network: www.lsneducation.org.uk 

Registered with the Charity Commissioners 

Copyeditor: Nick Sweeney 
Designer: David Shaw Associates 
Printer: Blackmore Ltd, Shaftsbury, Dorset 

052344RS/02/06/ 1500 

1-84572-413-5 

©Learning and Skills Development Agency 

You are welcome to copy this publication for internal use within your organisation. 
Otherwise, no part of this publication may be reproduced, stored in a retrieval 
system or transmitted in any for or by any means, electronic, electric, chemical, 
optical, photocopying, recording or otherwise, without prior written permission of 
the copyright owner. 

Further information 

For further information about the Learning and Skills Council, visit: 
www.LSC.gov.uk 

This publication results from the Learning and Skills Development Agency’s 
strategic programme of research and development, funded by the Learning and 
Skills Council, the organisation that exists to make England better skilled and 
more competitive. 



Contents 



Executive Summary 4 

1. Introduction 6 

2. What are the targets and indicators? 7 

3. The effects of performance indicators li 

4. The needs of the area 17 

5. Recommendations 23 

Appendix 1: Bibliography and Further Reading 27 

Appendix 2 - methodology and survey questionnaires 28 

Appendix 3 - the problems with performance indicators 30 

Appendix 4: the questionnaires used in the study 32 




Executive Summary 

1. This document reports on a research project held in 2005 which looked to see how 
performance indicators related to area needs. It shows the results of research 
across four LSC areas, with contributions also from national stakeholders and other 
witnesses. 

2. Performance indicators are regarded as useful by players at all levels of the 
learning and skills system - though for different reasons at different places. All our 
respondents felt confident about the validity of the figures in the core areas of 
recruitment and results. 

3. The implementation of performance management by indicators was however 
almost always a top-down process, with colleges and other providers striving to 
meet national levels in externally-defined areas of performance, and college teams 
having to meet demands set for them by senior managers. We found few examples 
where targets or performance indicators were used to support internal initiatives, 
and they were rarely seen as an interesting tool that could help understand a 
problem or yield information on contrasting approaches. 

4. It is often alleged that the usefulness of performance indicators is weakened by 
managerial or statistical manipulation: we found little evidence of this among our 
respondents. 

5. A new suite of success measures is being introduced by the LSC, which is being 
welcomed. Some elements are already in place: colleges specialising in Level 3 
work for 16-19 year-olds use value-added approaches widely. There is doubt as to 
whether value-added techniques can be extended fully to other clients and courses. 
Our respondents would welcome assessment of performance in provider families 
as another way to show their performance in context. 

6. Current quality measures work well for individual institutions, but have weaknesses 
in setting or assessing the needs of an area. This is because the attention is 
focused on the achievements of the people actually enrolled at the institutions, not 
on the population of the locality. 

7. The weaknesses are most evident in participation - especially, but not exclusively, 
at Level 2 and below. The problem is not caused by performance indicators as 
much as by corporate culture, where (for example) Level 3 work is 
disproportionately valued. However, the performance indicators in use do not 
highlight problems or show progress towards solving them. 

8. 14-19 area-wide inspections and Strategic Reviews have been used to analyse the 
appropriateness of local provision. However, despite evidence of good work in some 
of our areas, we generally found weaknesses in the mechanisms needed to secure 
the right balance of provision. Performance indicators, individual inspection reports 
and LSC systems seem focused on how well providers deliver, rather than what 
they deliver. 

9. Public Service Agreements (PSA) set the high-level government targets for our 
educational performance. But there are difficulties in establishing clear connecting 
rods between PSAs and local action. Given the importance of achieving the skills 
levels that have been set, more attention needs to be given to ways of delivering 
our national ambitions. The two essential elements are evidence-based local plans 
and an authoritative area-wide delivery system. 
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10. The local action plans we look for will need clear information, and an understanding 
of who is tasked to ensure their delivery. The LSC seems best placed for the job, 
but the lead role could vary between areas - perhaps being allocated on a 
competitive basis. Whoever is tasked with the job will need levers that encourage 
colleagues to make the changes needed. Opportunities should be created for local 
tuning of targets. 

11. The report ends with a series of recommendations concerned with new indicators, 
improved information, better local plans, clear local responsibilities, the creation of 
statistical families in the sector, improvements in management training, alignment 
of school and FE quality measures, and a greater focus on the nature of provision 
alongside its quality. 

12. Two important documents have been published since the completion of the 
fieldwork in this study. The Foster Review of Further Education (Foster 2005) draws 
attention to the issue of what it calls ‘locality’ - what we have termed ‘area needs’. 
The Annual Report of the Nuffield Review of 14-19 Education and Training 
(Flayward et al 2005) takes up this theme as it affects the younger age group. Our 
work is consistent with, and suggests ways to tackle, the analysis found in both 
these substantial documents. 
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1. 



Introduction 



1.1 This report gives an account of a research project undertaken under the auspices 
of the LSDA into the workings of performance indicators and targets in the learning 
and skills sector. The aim was to see how the widespread use of indicators affected 
the behaviour of institutions in the sector, and whether there was evidence that 
they caused the needs of some groups of learners to be less well met. Our 
particular focus was on the needs of an area - to investigate whether there would 
be advantages to be gained from setting performance indicators and targets across 
a geographical patch. Performance indicators as currently used tend to measure 
the learners enrolled in the institutions, rather than the whole local population. It is 
therefore possible for all the providers in an area to have strong performance 
indicators and inspection reports, while needs are not being met - employment 
sectors lacking trained staff, little choice for low-achieving youngsters, social groups 
disengaged, PSA targets unmet. We aimed to see if there was evidence for this, and 
if so, to establish why it happens and suggest ways to avoid the problem. 

1.2 The research took place between May and October 2005. A wide range of providers 
and relevant agencies were interviewed, with a particular focus on four LSCs and 
their providers. Our methods and approach are outlined in Appendix 2. We feel our 
findings are of considerable importance. On the one hand, we found little evidence 
of distortions caused by the widespread use of performance indicators and targets: 
they were accepted and generally used well, if a little narrowly. But they do not send 
the right signals as to what to do, because of the focus on individual institution 
standards. Our research confirmed that the current ways of contracting with and 
judging the quality of schools and colleges can leave a wide range of needs unmet. 
There are a number of reasons why government ambitions for educational progress 
- measured in the Public Sector Agreement (PSA) targets - are not being driven 
down to local delivery. We believe this lies behind the difficulties in meeting, for 
example, the Level 2 target for young people. At the end of this report we make a 
number of recommendations that aim to make it easier to hit the national targets. 
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2. What are the targets and indicators? 

2.1 It is important to be clear about the difference between targets, performance 
indicators, and data. 

■ Data consist of a number of observations of a measurable activity. When 
data are organised usefully, they become information. An organisation 
cannot collect information on every single aspect of its activity, but it should 
be able to measure its key activities promptly and accurately. 

■ Performance indicators use the data to create numerical standards that 
show how well key objectives are achieved. They aim to provide a yardstick 
forjudging performance - often by being associated with targets. 

■ A target expresses the desired value of an output at a particular time. If, for 
example, the performance indicator was the percentage of students who 
completed their course successfully, the target might be for 75% of those 
enrolled on November 1 st to remain on course and pass their chosen 
qualification in July the following year. 

2.2 Performance indicators and targets have become increasingly important in 
managing the UK’s private and public sectors in recent years. In the public sector, 
each government department agrees with the Treasury a suite of public service 
agreement targets (PSA) that represent the focus of their work. The attractions are 
as follows: 

■ They provide a form of accountability for substantial amounts of public 
money: the LSC will spend over £9bn in the current year. 

■ They allow elected governments to specify their priorities, and make the 
desired outcomes clear to those working for them. 

■ They can emphasise outputs and achievements for the client, rather than 
just activity or spending - the number of qualified young people rather than 
teachers employed, for example. 

■ They provide evidence about the effectiveness of alternative policies - for 
example, which approach is most successful in reducing unemployment, 
recovery from a given illness or engaging adult learners. 

■ They reveal differences in the performance of different institutions and 
agencies, allowing sponsors to put in place remedial intervention or 
changes in purchasing. 

2.3 These advantages do not just apply to over-arching agencies like the DfES or the 
LSC, setting standards for their suppliers. They can yield information to sharpen the 
sense of purpose, or assess and improve quality, volumes and value-for-money 
within institutions themselves. When we talked to providers, we tried to find out 
whether there was an internal culture of target-setting and performance 
management, and looked to see whether this was confined to passing down 
external demands or whether performance indicators were used to drive their own 
aims and standards. 

2.4 Performance indicators are not without their critics. It has been argued, for 
example, that social factors affect educational achievement so strongly that you 
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cannot truly compare institutions and areas; that managers can fiddle the books, or 
distort the picture of their organisation, to hit the numbers without improving the 
service, or that the cost of collecting the data diverts resources from the job in 
hand. We explain some of the perceived problems more fully in Appendix 3, having 
looked in our interviews for evidence whether or not these factors were coming into 
play. 

2.5 Although we make some comments later on management styles, the internal use of 
indicators is properly a matter for another study. Our main task was to look at the 
effects and use of government and LSC targets. As we have noted, the 
government’s expectations for the public sector are embodied in PSA targets which 
list the outcomes that the Treasury and the relevant department - in this case the 
DfES - look to deliver with the allocated spending for the forthcoming review period 
(Treasury 2004). There are two aspects relevant to the learning and skills sector: 
those for young people, and for adults. 

■ All young people to reach age 18 ready for skilled employment or higher 
education; this goal is made up of three distinct elements: 

by 2008, 60% of those aged 16 achieve the equivalent of 5 GCSEs (at 
grades A*-C), and in all schools at least 20% achieve this standard by 
2004, rising to 25% by 2006 and 30% by 2008 

increasing the proportion of 19 year-olds who achieve at least Level 2 by 
3 percentage points between 2004 and 2006 and a further 2 percentage 
points between 2006 and 2008, and increasing the proportion of young 
people who achieve Level 3 

reducing the proportion of young people not in employment, education or 
training (the NEET group) by 2 percentage points by 2010. 

■ Tackle the adult skills gap; increase the number of adults with the skills 
required for employability and progression to higher levels of training through: 

improving the basic skills level of 2.25m adults between the launch of 
Skills For Life in 2001 and 2010, with a milestone of 1.5m in 2005 

reducing by at least 40% the number of adults in the workforce who lack 
NVQ2 or equivalent qualifications by 2010; working towards this, one 
million adults in the workforce to achieve Level 2 between 2003 and 
2005 

by 2008, the numbers completing their apprenticeship to have risen by 
three quarters (that is, from 27% to 35%). 

2.6 The sector will also be an important contributor to achieving the higher education 
target: 

■ to increase higher education participation towards 50% of 18-30 year-olds by 
2010 . 

This is not just because the sector (if you include sixth form colleges) provides 
almost all the current crop of Level 3 graduates to enter the HE system. As nearly 
all the 18 year-olds with Level 3 qualifications already go to HE, increased 
participation will require the FE sector to help adult or non-traditional candidates 
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come forward to new forms of higher education, such as foundation degrees - 
whether young people from groups with historically low HE participation, the 
employed, or adult returners. One of our respondents spoke positively in this 
respect of the Aim Higher initiative, which is tasked with expanding entrants from 
urban areas. 

2.7 To help ensure that the PSA targets are achieved, senior policy-makers seek a ‘line 
of sight’ from national targets to regional to local institutions. The attraction is 
obvious: if it is possible to construct a clear set of connecting rods, we can make 
sure that national priorities are met. In some cases, this has worked well - our 
respondent from the Prime Minister’s Delivery Unit quoted success in reducing 
waiting times in accident and emergency units, and tackling adult basic skills. In 
other areas, success has been more elusive. For example, the target for 19 year- 
olds securing a Level 2 qualification was missed in 2002 and 2004. We look later 
at the issues that arise when seeking a line of sight in the wider education and 
training world, and how they contribute to problems hitting ambitions for Level 2 
attainment, for example. 

2.8 National LSC officers attach great importance to the PSA targets. In our 
discussions, they stressed how the targets set for regional and local LSCs were 
derived from the PSA and demographics. Local LSCs are keen to meet the targets 
set by the new regional management structure. However, although they reported 
that they take account of the PSA targets, they are not formally delegated ‘down the 
line’ to providers. In many ways this is wise. Educational outputs cannot be ordered 
like so many apples or cars. They depend on a mutual transaction between the 
learning institution - a school or college, perhaps - and a learner. Training more 
plumbers - to take a contemporary example - requires not just colleges and other 
trainers to offer the places. 1 It requires people to come forward to take them up, 
and to follow the course to a successful conclusion - which itself can be affected by 
changes in examination requirements (such as key skills). Further, there are 
outputs and activities beyond the bare PSA outlines that the LSC and DfES wish to 
see from their providers - such as better-trained staff and improved engagement 
with employers. 

2.9 As a result, LSC-funded providers currently have to meet a suite of performance 
indicators derived from the ‘Success For All’ White Paper of November 2002. These 
were described to us as ‘the big four’: 

■ student volumes 

■ student success, a composite of 

- retention (or its converse, wastage) 

- achievement (pass rates) 

■ staff professional qualifications 

■ employer engagement. 

Many of our respondents viewed these as separate endeavours, but others - 
rightly, in our judgement - saw them not as separate boxes to tick, but as a suite 
whose elements tended to impact on one another. They argued that better links 
with employers aided recruitment and success, or that well-trained staff boosted 
results. 



1 Itself no simple matter, as construction staff are difficult to recruit when the industry is buoyant (and why the FEFC/LSC 
funding has depended on current student volumes making it difficult to keep capacity in place when - as in construction - 
trainee recruitment is cyclical). 
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2.10 In the summer of 2005, following an extended process of consultation, the LSC 
announced a move to a broader and more sophisticated range of measures of 
success. 2 These are described in the document ‘Implementing New Measures of 
Success’, and comprise: 

■ qualification success rates 

■ value-added and distance travelled 

■ recognising and recording progress and achievement (RARPA) in 
non-accredited learning 

■ responsiveness to employers 

■ learner satisfaction 

■ learner destinations 

■ value for money 

■ staff measures. 

The new measures are explained fully on the ‘Success For All’ website. They aim to 
allow a broader assessment of provider performance, and will have the advantage 
of reducing the incentive to cream or adapt admissions standards, thus allowing a 
fairer assessment of the effects of disadvantage. It will be noted, however, that the 
new measures do not have a participation element - either in student volumes or 
in diversity or level. Nevertheless, our interviewees looked forward positively to the 
new measures, despite some scepticism about the difficulties involved in making 
some of the assessments. A substantial programme is under way to introduce and 
implement the new measures, but during our research both LSCs and their 
providers were working to the older four headline measures. 

2.11 As our work progressed, we found that it split into two main themes. First, what 
exactly are the effects of performance indicators on institutional behaviour? We 
looked to see how the numbers impacted on organisational purpose, whether 
managers saw performance indicators as a key tool in their task of raising 
participation and standards, or a tedious set of forms to dispatch to a distant head 
office. We particularly sought evidence on the grounds of perverse behaviours that 
make the achievement of real improvements illusory. The second element of our 
research brought in an area dimension. We looked at the question of whether the 
needs of the local population were not being met, even if every local school and 
college were - according to inspectors or league tables - doing well. It has even 
been alleged - for example, by one of our respondent LSCs - that the two 
questions could be negatively linked: that institutions ignore social provision that 
will reduce their league table score. These questions are addressed separately in 
our next two sections. 



2 Details and helpful explanations of the new measures are to be found at: 

http://www.lsc.sov.uk/National/Documents/Kevinitiatives/SuccessforAII/nms-background-documents.htm (accessed 
11 / 01 / 05 ) 
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3. The effects of performance indicators 

3.1 We asked all our respondents their overall views on the increasing use of 
performance indicators to measure and manage their work. Some national 
respondents - NATFHE, NIACE and the Secondary Heads Association (SHA) - had 
considerable concerns about the issue. They worried about the well-known 
drawbacks of performance indicators and targets, such as social context and 
perverse outcomes. 3 These reservations were shared by two of the LSCs we spoke 
to as well. But when we spoke to the providers themselves, we found a different 
story: widespread acceptance of the usefulness and validity of performance 
indicators and targets, and a feeling that to cheat the figures - even if it were 
possible - would be self-deluding. This was reported across the full range of users, 
particularly in the assessment and development of quality, and at all levels of their 
organisations. Few respondents reflected the dissent of previous years - that this 
mode of management was an intrusion into professional decisions, or that social 
factors made the system unfair and inaccurate. The vast majority knew of their 
performance standards, regarded comparisons and indices of performance as 
helpful, and had strategies to improve weak areas. 

3.2 This does not mean that our respondents were unaware of some of the problems 
with the uncritical use of performance indicators. Many spoke of the need to 
recognise social deprivation. This was keenly felt in an urban FE and an inner-city 
sixth form college we spoke to, but included those, like the senior manager at a 
sixth form college and a middle manager at a Beacon tertiary college, who were 
conscious that they worked with easier target groups or environments. A number of 
our respondents said that they would welcome benchmarks of comparable 
institutions that would give more genuine assessments of performance and allow 
them to see what it would mean to become ‘best in breed’. The other - telling - 
attraction to middle managers of grouping similar institutions together for the 
purpose of comparison is that it would help convince sceptical staff that improved 
performance is possible, even in their context. Acknowledging the importance of 
social factors, however, did not lead to any excuses, even from institutions plainly 
working in challenging environments. 

3.3 It could be argued that the increasing acceptance of a suite of performance targets, 
often handed down from above, is a sign of increasing compliance by the sector. 
However, previous LSDA research - for example, on performance-related funding - 
has shown that vigorous opposition continues to be expressed to measures that are 
seen as unfair or wasteful. What we did find, however, was that the ways of using 
performance indicators varied between providers. Those who were vulnerable 
spoke of the desperate need to move their measures upwards - staff at one LSC 
spoke of the utility of targets in ‘stiffening the resolve’ of the providers they contract 
with. More confident and successful institutions saw numerical measures as a 
useful tool in continued quality improvement, or in confirming their success. There 
was, however, little sophistication in the understanding of performance indicators; 
for example, none of our provider respondents were familiar with the curriculum- 
adjusted success rates that will be increasingly used to assess institutional 
performance. This is important - a given portfolio mix can make a performance of 
60% in one college, for example, better than a 70% benchmark in another. 4 



3 There is a discussion of the theoretical problems with performance indicators in Appendix 3. 

4 This derives from the fact that different programmes, subjects and levels have different pass rates. To take an example, 
A-level pass rates are usually around 90%, whereas GCSE A-C grades come in below 60%, so a sixth form college which 
introduced a GCSE stream would see its overall success rate go down for that reason alone - with no deterioration in 
underlying performance. Even within a particular programme there are variations: for example, twice as many students get 
grade A in A-level Maths as in Psychology. 
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3.4 There was an understanding of the difficulty of translating some national ambitions 
- expressed in PSA targets - into targets for individual institutions. Some national 
goals - such as reducing the numbers of young people not in employment, 
education or training (the NEET group) - are not wholly within the power of the 
educational sector to deliver, because of the employment dimension. Others - the 
aim for 50% of the age group to enter HE, or the targets for Level 2 attainment - 
refer to a whole cohort rather than the entrants to a particular college or sixth form. 
At worst, we encountered examples where respondents supported the 
government’s goals, for Level 2 or the NEET group, but felt it was someone else’s 
job to work on them. 

3.5 None of our sample reported major difficulties in generating accurate data. Most 
respondents saw the ability to produce accurate data as an important tool for 
planning and quality improvement. They saw it, as do the LSC and inspectorates, as 
a sign of competent management in itself. 5 There was some frustration at the 
delays inherent in current management information systems, which obliged 
managers to look to previous years for data on results, for example. However, some 
of our providers were able to offer more up-to-date data to support campaigns to 
raise punctuality or attendance levels. Colleagues in the sixth form sector found 
that the coming of performance-related pay for teachers helped the drive for 
accurate figures. Another plea was for consistent figures through time to support 
judgements and action on internal quality. This is something to bear in mind as 
national measures change: managers, like scholars, need continuous series. 

3.6 Paragraph 2.9 listed the ‘big four’ indicator areas. We found that the difficulty of 
meeting these measures fell differently on different institutions. For example, sixth 
form colleges and schools had little difficulty in achieving the required levels of staff 
teacher qualifications - indeed, until recently it was a legal requirement for a 
secondary teacher to be trained. However, some of our general further education 
college sample had been obliged to work hard to make progress: they are more 
likely to use agency staff, or have teachers and trainers entering directly from 
industry without teacher qualifications. An additional difficulty is caused by staff 
mobility - which meant that a college which had successfully trained a lecturer who 
moved to a post elsewhere would lose them from their statistics. Indeed, one of our 
LSC respondents doubted whether the data on staff qualifications were 
worthwhile. 6 There was also a widespread understanding that ‘employer 
engagement’ was desirable but - despite the current efforts of the DfES and LSC - 
difficult to measure. 

3.7 The acceptance and usage of performance indicators by our respondents was not 
always accompanied by a sophisticated understanding of their strengths and 
weaknesses. Many in our sample used the terms ‘performance indicator’, ‘data’ 
and ‘target’ interchangeably - answering a question about internal performance 
indicators by assuring us that they did indeed gather data on ,for example, student 
attendance, staff qualifications or diversity in recruitment. But senior staff at the 
LSC national office said that it was important to distinguish between targets (things 
to hit) and performance indicators (useful information). A local LSC told us that 
what was important was the dialogue around the performance indicators, a view 
shared by NIACE. This was in contrast to the view from the provider. Almost all our 
respondents felt that it was crucial for their institution or section to achieve a 
benchmark level. Few used indicators as they were supposed to be used - to 



5 This is a further reason to doubt stories of widespread manipulation of the figures; to do so would require a college to be 
really on top of its management information - and generally this sort of institution has no need to massage its figures. 

6 It can be argued that the LSC should not be concerned with staff qualification rates in their providers; their job is to buy 
good provision, the providers’ job is to supply it however they may. The link between quality and teacher qualification, 
however intuitively attractive, is in any case quite weak. 
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assess progress, or suggest areas for further investigation. And despite an 
understanding that performance indicators are only useful where they can be 
linked to change, there was little evidence of a close connection between desired 
targets and actions taken to move towards them - for example, an idea of how 
many percentage points retention might move from a given change in student 
guidance, or the size of the effect on success rates of a change in awarding body or 
course structure. Yet such judgements are important in setting a reasonable target 
- which college governors are obliged to do - and choosing between alternative 
strategies for improvement - which is what managers must do. 

3.8 All our respondents were aware of the importance of hitting their targets as an 
institution, and most devolved some targets and performance indicators down to 
sections. This was nearly always concerned with the LSC’s suite of indicators - 
primarily those on recruitment, retention and achievement. Generating internally 
chosen performance indicators was rarer, though some sixth form and tertiary 
colleges were keen to set targets for attendance and (as we shall see) value added. 
These, of course, are factors that feed into the current LSC performance suite: 
attendance is closely related to retention 7 , and an understanding of value added 
can help strengthen results. Some of the larger general further education colleges 
had developed a more thoroughgoing style of target-driven management. At least 
one large college used data and targets to assess performance in everything from 
library issue to technician utilisation. Another checked target class size and staff 
utilisation, a third linked senior managers’ pay to target attainment, which extended 
to ethnicity, and the financial contribution of courses. This was exceptional, 
however. And while a few respondents told us they sought information about 
diversity, generally we found no evidence of schools and colleges using 
performance indicators more broadly as a way to manage, for example, curriculum 
changes, progression or widening participation, or to even to meet the demands of 
the Disability Discrimination Act, which will oblige them to set and monitor targets. 

3.9 We found instances of devolution of targets for staff qualifications and employer 
engagement, but the main focus was on recruitment and results. Section heads 
were unanimous in agreement that numerical measures were essential to identify 
strong and weak performance, and to assess the progress of changes. Benchmarks 
and targets seemed particularly forcefully used where newly-appointed managers 
felt the need to make a step change in standards of delivery. But what tended to 
happen was that performance indicators were used as a stick to beat under- 
performing areas, rather than a tool to identify problems and ways that might be 
appropriate to remove them. So what we saw was performance indicators being 
used differently at different levels - at national level to secure government targets, 
at LSC level to contrast the performance of providers, at provider level to assess 
progress and at departmental level to drive quality improvement. 

3.10 Generally, the use of numerical drivers reflected the style of the organisation and, 
to a lesser extent, its size. A few of our respondents did not translate external 
performance indicators into internal action at all. A high-performing Midlands 
college and a well-regarded sixth form college both argued that getting the systems 
and actions right would deliver strong outcomes without internal targeting. One 
claimed that in a small provider, the management and governor scrutiny was 
sufficiently rigorous not to need these prompts. Another argued, interestingly, that 
once you devolved targets to departmental heads you would be obliged, in fairness, 
to devolve staffing and budget decisions: and that this would move their 
organisational style from a whole-college approach to individualism - in a way that 



7 Research suggests that students rarely drop out suddenly - the most common predictor of wastage is ever-worsening 
attendance. 
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they did not want. This contrasted strikingly with the response of a London college 
that welcomed the element of internal competition generated by the performance 
records of different departments. 

3.11 We were keen to see if there was any evidence of the criticisms of performance 
indicators mentioned in Appendix 3. Our questions sought out any information 
about how institutions or individual sections within them might indulge in gaming 
behaviour. We tried to tease this out with this question for middle managers: 

Performance numbers can be improved in various ways - improved 
teaching/changing staff, setting targets for improvement, improved student 
support, changing to more appropriate courses and assessment, moving to 
more short courses, better admissions guidance, stricter exam entry, counting 
the figures differently, better equipment or marketing. Can you give an example 
of how you have improved performance in your section - which of these 
strategies (or any others) did you use? 

There were a few indications of institutions taking administrative actions to ensure 
they hit their recruitment - and thus budget - targets. One college had moved its 
adult education programme onto FE-style courses. Elsewhere, a local LSC worried 
that, for budget reasons, local school sixth forms retained students to the PLASC 
count date 8 but no further. 9 But we did not see any indication of gaming activity 
when it came to quality and results - indeed, managers told us that they were 
vigilant to ensure that it didn’t happen. This is not to deny that policy changes had 
been made that improved the indicators - for example, the introduction of modular 
programmes, or more austere admissions guidance. This was also noted in the York 
Consulting study on improving colleges. 10 But there, as here, staff were keen to 
point out that the changes were introduced to benefit the students, not to tweak 
league table positions. Respondents argued that it was not in a learner’s interest to 
take an over-ambitious programme, and motivation was improved by breaking 
assessment into chunks. 

3.12 One criticism of current performance indicators is that they do not measure the 
distance travelled by a student, only their final attainment. This can work in favour 
of institutions and departments teaching very able entrants, and against those 
working in tough inner-city locations. It is often claimed that this is the reason that 
sixth form colleges are over-represented in the Learning and Skills Beacons and 
recipients of performance-related funding, 11 for example. To get around the 
problem, there is increasing interest in the use of value-added techniques for 
measuring educational success. Value added (VA) measures look not at the level 
and grade of the final attainment of a student but at the distance travelled, based 
on their entry qualifications. We found substantial use of VA techniques in sixth 
form and tertiary colleges where the main client group was full-time 16-19 year- 
olds doing Level 3 programmes. Where VA techniques were used, departments 
were judged on how successful they had been in adding to student educational 
attainment in their stay. This has the advantage of providing an honest assessment 
of the quality of pedagogy - avoiding the ‘creaming’ problem, as there is no longer 
a benefit in performance indicators from establishing over-high entry qualifications. 



8 Student retention in colleges is checked at three census points in the year. By contrast, schools simply have to retain 
their students until a single date in mid-January. 

9 This will reflect badly on a school when the measures of progression proposed in the 14-19 strategy are delivered. These 
suggest that schools should be accountable for the staying-on rate of ajl their pupils - not just the sixth formers. 

10 This is as yet unpublished research work commissioned by the DfES. It should be noted, though, that York Consulting’s 
sample was made up of colleges which had achieved strong underlying improvement - leaving out those whose numbers 
had improved due to, for example, running more short courses. 

11 Performance-related funding is a recent innovation that provides a funding uplift for colleges with results well above the 
national average: it is now proposed that it be replaced with an award for fast-improving providers. 
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3.13 We saw earlier (para 2.9) that the LSC plans to extend the use of value added in its 
assessments of provider quality, and has published a number of proposals that 
offer the chance of extending the well developed GCSE/A-level value-added 
schemes to vocational work. There remain a number of intractable problems with 
the wider use of value added, however: 

■ Returning adults often have no reliable baseline from which to judge 
distance travelled. Many have no qualifications at all, and even if they do, a 
qualification from 20 or 30 years ago is rarely a good guide. 

■ NVQs are typically ungraded. Being competence-based, they are simply 
pass/fail, and so a scraped pass will score as well as a superb result. 

■ Many students study at the same level as a qualification they already have 
- for example, a graduate nurse or engineer looking to gain a management 
qualification, a plasterer studying electrical installation. It is difficult to 
argue that they are making no gain in learning. 

■ There are various issues concerning ESOL learners. 

Value added is also unlikely to be able to address the next issue we will discuss - 
that is, seeing how well the balance of provision in an area fits the needs of the 
whole population. Like conventional measures of success, value added only looks 
at those who are enrolled at an institution, not those for whom provision is lacking. 

3.14 One problem with gaining the full benefits of our system of indicators and targets 
that was presented to us was the rate of change in the policy environment. Even 
where there was consistency in the goals, the level of importance changed through 
time. A number of respondents commented on this - that student growth, once so 
important, was now subordinate to quality, 12 how widening participation was now 
‘on the back burner’ while employer engagement had gained greater importance. It 
is important for national stakeholders to be aware of this feeling, in case focus is 
lost as those at the delivery end feel that, like a bus, another key priority will be 
along in a moment. We were struck by the number of respondents who were aware 
of a problem - with planning, funding, bureaucracy or quality development - but 
felt that they did not need to do anything about it because it would be sorted out in 
the coming round of reports or policies. This is understandable, given the plethora 
of activity. Current developments include the : 

■ ‘Success For All’ strategy 

■ Skills Strategy volumes 1 and 2 

■ 14-19 White Paper 

■ Sweeney Reports 1 and 2 

■ Foster Enquiry 

■ annual DfES grant letter 

■ LSC’s Agenda for Change 

■ Leitch Report 

■ Tomlinson Report 

■ DfES Five-Year Plan 

■ National Audit Office Report on Strategic Leadership in FE. 

No public service linked to the fast-moving employment and industrial scene can 
stand still. However, it’s important to minimise the opportunities for the excuse we 
sometimes found - where respondents said that they knew of problems but they 
would be addressed in time by the LSC, the DfES or the inspectorates. PSAs have 



12 This reflects the downgrading of the participation in PSA from a hard number for 2002 to an aspirational target for 2004 
- perhaps because increasing volumes can more plausibly call for additional resources than improved quality. 
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been consistent for a number of years, and even elsewhere the policy thrust is in 
fact reasonably coherent - raising quality, increasing choice, improving 
participation and simplifying planning - and can be progressed at all levels in the 
sector. The rate of change is fast, but that should not be allowed to provide an 
excuse for local leaders to abdicate responsibility for addressing key local issues, 
which do not change substantially from year to year. 

3.15 To summarise where we have got to, we found that performance indicators were 
now broadly accepted as a way of comparing providers, and as a way of supporting 
quality improvement. There is no evidence that performance indicators are 
resisted, or fiddled. But we also found, as we shall see, that many providers do not 
shape their provision to meet local needs. We started out with the aim of seeing 
whether this was done in order to look good in league tables, but found little 
evidence of it. So if performance indicators are not driving institutional choices, 
what is? We discussed with our respondents many other determinants of 
institutional behaviour besides performance indicators. Funding was mentioned by 
most. Achieving the recruitment target was always presented to us as important 
because of its effects on the college budget, and never as a measure of raising 
local skills, or credibility with the community. As far as quality goes, Ofsted/ALI 
inspection was named as a powerful driver. LEAs reported that they had real 
credibility - more so than performance indicators and league tables. Overall, we 
were struck by the importance of institutional culture, particularly the valuing of 
academic and higher-level programmes. We shall return to this in the next section. 
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4 . 



The needs of the area 



4.1 The second aspect of our work looked at whether performance indicators might 
distort provision in an area - to explain any mismatch between the local offer and 
the needs of the population. Our interviews confirmed disquiet about a mismatch 
between what institutions were offering and what local areas needed. This has 
been highlighted in a number of Ofsted Area inspections - where institutions 
received good grades for achievement but the area scored poorly for access and 
participation. Even outside the area-wide inspection areas, we learned of districts 
with low participation even where local institutions are strong, even, in one case, 
Beacons. This has the effect of restricting choice for a wide group of learners - 
working against the major government objective expressed in theme 1 of the 
Success For All strategy. To make choice effective, it is not enough for there to be a 
range of places to study - there must be an offer that meets learners’ needs. 

4.2 There was a particular concern about the balance of high-level and low-level 
provision for young people. Young people who gain five ‘good’ GCSEs had a good 
choice of options everywhere we looked - either the academic route to A-levels, or 
their equivalent vocational award. Those who do not achieve that benchmark - and 
this is more than half the cohort in many areas of the country - can find their 
choices much more restricted. We found much evidence to support the views 
expressed in the Chief Inspector’s Annual Report: 

As in previous years, the most able young people at 16 continue to have the 
best choice of courses and often the best quality of provision post 16... in most 
areas there are still insufficient courses at intermediate and lower levels to 
meet the needs of the less able 16 year-olds... (Ofsted 2004a, paras 212 & 
213) 

Analysis of a number of 14-19 area reviews provided ample confirmation for this 
view. One of our own respondents reported as follows on their study of an area full 
of high performing providers: 

Institutional measures are either good or show major improvement, but local 
participation rates have hardly moved and are dire in the poorest areas - and 
there is a yawning curriculum gap for those who have not achieved a Level 2. 
Low area performance is no-one's responsibility. 

4.3 The issue was evident in the discussions about the national failure to hit the PSA 
target for Level 2 attainment at 19. One LSC told us: The market caters for Level 3 
very well.’ Another added concerns about the lack of provider enthusiasm for 
meeting the needs of disengaged young people: ‘We now need a push for Level 1 
provision.’ These remarks refer specifically to provision for young people, but the 
problem does not stop there. The Skills Strategy’s supporting papers show 
matching issues in respect of adult provision. One in five vacancies proves difficult 
to fill due to skills shortages; 40% of employers reckon they lose business because 
of lack of the right skills. 

4.4 Do our current performance indicators contribute to the difficulty in meeting the 
needs of all the learners in an area? There are a number of reasons why that might 
be the case. Our survey has found no evidence for the crudest one - that providers 
would alter their programmes or ignore needs because that was the easiest way to 
achieve a high placing in league tables. No one told us - and remember, this was 
an anonymous survey - that they decided to emphasise Level 3 provision, ignore 
employer needs or avoid low-level work because of the effects on performance 
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indicators. 13 However, the reaction of a number of respondents to provision that 
they had found hard to improve was to withdraw from it. This leaves the possibility 
that important work is abandoned because it is difficult to do well - or even that we 
underestimate how difficult it is to do some work well. It is significant that many of 
the institutions excoriated in ‘Why Colleges Fail’ (Ofsted 2004b) specialised in work 
at Levels 1 and 2. 

4.5 It was not the institutional performance indicators that prevented providers from 
meeting local needs fully. 14 We believe the problem stems from two reasons. The 
first is an issue of culture. There seems to be a widespread feeling that Level 3 
work is more desirable - that an institution should prioritise this work above others. 
It was never stated, but is implicit. We felt that it was this that prevented sixth form 
colleges, for example, moving to close the gap of Level 2 attainment at 19, rather 
than any concern about their league table position. More than one respondent felt 
that admitting students below Level 3 would create difficulties, worsen the college 
atmosphere and lower its reputation. One college in a deprived area bussed 
students in from a distance so that it could maintain its standards of academic 
entry: it justified this to retain high levels of recruitment without considering 
alternative strategies for other cohorts of learner. Elsewhere, a Connexions 
respondent worried that a large and well-regarded college had made a strategic 
choice of going for the academic work with young people, with the result that 
disadvantaged and disengaged learners had to be accommodated in less well- 
resourced and regarded community trainers. Both of these colleges have recently 
received enthusiastic inspection reports, and stand high in performance league 
tables. 

4.6 The cultural attitude was not generally due to callousness - though one institution 
justified its stance by claiming: ‘We work for students, not national politicians.’ The 
institutions we spoke to saw the PSA Level 2 target as important, and supported 
the drive to reduce those not in employment, education or training (NEETs). 
However, many felt it was not their job to remedy the problem. In one striking case, 
we interviewed school sixth form leaders who felt that a newly-established sixth 
form college was about to tackle this task: but in interviewing that college, we found 
that it had no such intention. The cultural question seems particularly acute to us, 
for it is not just staff and governors who regard Level 3 work as their flagship 
provision. Parents look favourably on colleges with high pass rates and university 
entry, and local newspaper league tables - mentioned by more than one 
respondent - regard raw A-level scores as the touchstone of quality to the exclusion 
of almost all else. Changing the position would seem to call for considerable 
encouragement and leadership - backed perhaps by financial incentives. 

4.7 We did find some instances of providers reshaping their mission to better meet 
local need. One general further education college was considering leaving A-level 
work to local sixth forms, another took over a local work-based learning (WBL) 
operation, a third had worked with its LEA to rationalise local adult provision. In the 
Midlands, we saw a major provider moving into new niches to secure its 
recruitment numbers in a vigorously competitive climate. These changes are driven 
by an assessment of local need, and an eye to hitting recruitment numbers, rather 
than being part of an LSC plan or a reaction to measured quality of performance. If 
anything, the decisions would make the college performance indicators apparently 
worse. But if we found that performance indicators do not create the problem, they 



13 There is a possibility that funding pressures would move institutions towards Level 3 work, which is better resourced 
under the current funding system. However, this is not widely known, and we found no evidence of it. 

14 However, one LSC (Birmingham and Solihull) did feel that schools and colleges seeking league table security made it 
more difficult to hit their inclusion goals. 
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are not mobilised to help. Colleges were aware that the performance indicators 
measured the service for those in the college, not the people in the area. Even in 
areas with well-developed collaborative arrangements, performance indicators were 
seen as a way of measuring quality between providers. The new measures of 
quality outlined in para 2.9 may make this task easier for LLSCs - and perhaps 
help the leaders of those providers to assess their own progress better - but they 
will do little to change the focus of local provision. Nor will the current levels of 
funding - which are more generous for a broad full-time A-level programme than 
almost any other course. 15 

4.8 If culture is important, so is organisation. The second reason for failure to meet 
area needs is that we have not yet developed a clear way to identify area needs 
and connect them to forceful delivery. When we asked our respondents about the 
needs of their area, we often received thoughtful and well-informed answers. LSC 
plans often reflect a considered and comprehensive understanding of area labour 
or social needs, but these are rarely driven down to provider behaviour. College 
strategic plans can also make reference to the local community’s requirements in 
its needs analysis. Again, however, the links that were made between those 
assessments and the delivery plans were seldom strong. For colleges, programme 
planning is usually incremental, based on tuning the programme in the light of the 
previous year’s volumes and quality. Some LSC staff we spoke to were aware that 
quality was not enough - that they needed to rebalance local provision. However 
they, and Connexions and LEA staff, felt that they lacked the tools to engage with 
the problem. We found a particular weakness in directing the strategic contribution 
of sixth forms. 

4.9 The problem of addressing area needs has been tackled in a number of ways by 
policy-makers. The first instrument was the 16-19 Area-Wide Inspection 
mechanism, which soon became 14-19 inspection. These were driven by Ofsted, 
and lie behind the remarks quoted earlier by the Chief Inspector. They have had a 
positive effect. We were encouraged to hear that Area-Wide Inspections often had a 
‘galvanic effect’ when all the partners - Connexions, WBL providers, LEAs, schools 
and colleges - came together to make their self-assessment report, often for the 
first time. Each review was followed by an action plan aiming to remedy any 
weaknesses found in the analysis. The LSC’s report on follow-ups to Area Inspection 
(LSC 2005) seems to us fair when it says: 

In contrast with earlier reports where there was a greater emphasis on 
structural reorganisation... the emphasis is more on reshaping the 
relationships between existing providers. 

And, while praising many worthwhile developments after area inspection, there is a 
repeated note of caution in an assessment that: 

...learner entitlements... can only be delivered if the strategy ensures learners 
ready access to courses best suited to their past achievements, aptitudes 
and aspirations, and to local skills and community needs. 

We found our respondents were aware of Area Reviews and their findings, but the 
action plans seemed rarely to impact on them at a practical level. The 14-19 area 
inspection system is being supplanted by a wider Joint Area Review (JAR) starting 
from the age of 3. Some of our respondents were worried that this broader 
approach will weaken the focus needed to get post-16 provision right. 



15 Level 2 is slightly less well funded than Level 3, generally reflecting fewer taught hours, but this does not seem to have 
been a driver - or even known - to our respondents. 
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4.10 The ‘Success For All’ reforms of 2002 introduced another mechanism for matching 
local provision to need in the Strategic Area Reviews (StARs) that all local LSCs 
were required to complete by March 2005. The approach taken to this task differed 
from LSC to LSC. Some chose an area focus, while others looked at provision by 
age range, by level or by industrial sector: many mixed their approaches. The 
National Audit Office may be right when it says that ‘these reviews... have helped to 
start a change in the culture in which people and organisations are becoming more 
willing to collaborate to reduce collaboration and fill gaps’, but our respondents 
rarely, if ever, referred to StARs. Moreover, we found few cases where there had 
been a substantive follow-up. This may reflect risk-aversion. We had comments 
from an LEA arguing for rationalisation, and from an LSC saying ‘infrastructure 
matters’, but there have been some well-publicised cases where rationalisation 
arising from StAR has been rejected - as in the Hastings case that led the Guardian 
to conclude that the strategic review process was a ‘dead duck’ (Kingston 2005). 
We would not go that far, and found examples of effective collaboration - but these 
rarely seemed strong enough to create the right balance of local provision. 

4.11 So the analysis of local need by LSCs and providers does not lead to effective 
action, and the 14-19 reviews have yet to secure the balance of provision needed. 
It is important to realise that this is not the fault of LSCs or LEAs. The Chief 
Inspector’s Report comments on the lack of a clear statutory framework for 
developing 14-19 education, and draws attention to the tensions between 
corporate autonomy and local planning: 

The influence that they [local LSCs and LEAs] can exert on local planning is 
limited by the considerable autonomy of individual schools and colleges, 
particularly in the absence of incentives for strong sixth forms to engage in 
collaboration. Consequently the LEAs and local LSCs have not yet brought 
about sufficient improvement in standards and levels of participation for 14- 
19 year-old students. The available management information is usually 
inadequate to inform planning. 16 (Ofsted 2004) 

4.12 One arm of government delivery for 14-19 is collaboration via partnerships. In our 
research we found that these varied greatly in strength and importance. In some 
areas we found strong and cohesive networks that were plainly capable of 
delivering local priorities: area targets could confidently be contracted to them. 
Much more common, sadly, was a ‘strategy vacuum’ that makes national PSA 
targets very much more difficult to achieve. In one city the LSC said that strategy 
was the job of the colleges, whereas the provider respondents said it was the job of 
the LSC. This may be an extreme, but it reflects a phenomenon noted by the 
National Audit Office (NAO 2005) that ‘there are now many more influences on 
colleges’ strategic directions than previously and these have eroded colleges’ 
perceived autonomy to set their own strategic direction’. Perhaps the ring can be 
held by local learning partnerships, but we found many respondents regarded them 
as talking shops: in one northern city, the sole college was aware that a partnership 
had been set targets but didn’t know what had become of them. At the extreme, a 
large sixth form we spoke to did not even know of any partnership activity at all. So 
firstly, we lack a delivery arm with power. 



16 This comment contrasts with the view expressed by the LSC study of area review (LSC 2005) which claims that ‘in all 
areas high-level strategy has been underpinned by operational arrangements to maximise choice and minimise 
duplication... underpinned by agreements to share data on recruitment, retention and achievement across an area’. Our 
conclusions suggest this is optimistic: we side with the Ofsted view. 



20 




4.13 It would not, in any case, be an easy task to connect PSA targets to provider activity 
even if there were effective local delivery systems. We came to the view that 
creating the clear line of sight sought by policy-makers - from national PSA to 
institutional delivery - is inherently difficult. 17 It was a view shared by our 
respondents, one of whom described the whole idea as ‘implausible’. There are a 
number of reasons for this: 

■ Pre-16, the whole age cohort goes to schools: targets are easier to 
construct and monitor across a local education authority area. But post-16, 
attendance is not compulsory. Many young people drop out of education at 
16 or 17, and adult participation in lifelong learning is patchy. There is 
therefore a dissonance between the population at large - for whom PSA 
targets apply - and the population in institutions - to whom performance 
indicators and LSC allocations refer. 

■ The 16-19 infrastructure differs greatly from place to place - in some 
areas, 11-18 schools dominate the scene, in others 11-16 schools feed 
sixth form or tertiary colleges. LEA commitment to community education 
varies, and so does delivery systems: with discrete community education 
services, or delegation to local colleges. FE can be general or specialist, 
with a community or vocational bias. Institutions differ in their client groups 
- a work-based provider is not concerned with HE entry, nor is a sixth form 
college concerned with NEETs. And there are often complex patterns of 
travel, with many local people studying outside their area of residence. In 
this situation, it isn’t simple to allocate targets down the line to a local 
delivery system. 

■ Some goals, like NEETs, go beyond the learning and skills system. The 
numbers in employment - part of the measure - are not within the gift of a 
local LSC or partnership. The ability to hit the HE target will be affected by 
the ability of local schools to provide candidates ready to succeed at Level 
3, or by the practice of the university sector. The numbers going through to 
HE are in any case strongly affected by social factors, and so ‘line-of-sight’ 
would seem to require context-sensitive targets, community by community. 

■ PSA targets concentrate on outcomes, not activity. As a result, they depend, 
for example, on trends in retention and attainment as well as volume. And 
as we saw earlier - para 2.8 and footnote - even the right level of funding 
can’t even guarantee the right number of course starts, as that depends on 
demand. This creates a number of ways that the target might be missed: 
inadequate budget allocation, low recruitment, retention below that 
expected, poor pass rates. 

■ As we have seen above, LSCs do not have the power to drive institutions: 
the Chief Inspector points out that ‘an effective strategy framework is 
lacking for analysing the needs of young people and the community across 
an area and designing the provision to meet them’. There are current 
moves to improve the linkages between LSC allocations and government 
priorities, but these do not seem to be based on local fine-tuning of PSA 
targets. 



17 There is an exception: floor targets are easy to delegate. For example, it is much simpler to insist that no school falls 
below 20% of pupils getting 5 ‘good’ GCSEs than to ensure that 50% of the nation's young people have an experience of 
higher education. 
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4.14 These issues came to the fore when the Prime Minister’s Delivery Unit conducted a 
review of the Level 2 target for young people. This was missed in 2002 and 2004, 
and the DfES and Prime Minister’s Delivery Unit undertook a joint review to ensure 
that progress would be made for the 2006 and 2008 targets. Delivery on an 
acknowledged problem isn’t the only issue: it can be a problem sometimes to know 
where we are. How do we find out whether our Level 2 target, for example, has 
been hit when tracking attainment between school, college and workplace, amid a 
plethora of qualifications ? Until recently, this could only be assessed by a self- 
reporting question in the Labour Force Survey. This position looks likely to be largely 
solved by the introduction of ‘the admin data set’ - a database gaining information 
on qualification levels from matching the records of providers and awarding bodies. 
This innovation needs to be pressed home locally, where it will be enormously 
helpful. 

4.15 Clear data - driven down to localities - can help us define the task, and 
empowered local networks can allocate it to institutions. This will require a decision 
about ‘who does what’ - who will deliver the locally-needed Level 2 places, for 
example, or outreach to the NEETs group? These questions are clearly linked to 
current debates about the mission of colleges, about specialist schools and 
vocational centres of excellence. A place needs to be found also for private sector 
WBL providers. It is particularly important to confront existing notions of worth in 
post-16 education: we were told repeatedly that there was no problem finding 
quality places for able Level 3 candidates. If we are to hit the PSA targets for Level 
2 and for adult basic skills, high-quality providers must be encouraged to contribute 
to that strand of work. There is a similar issue in neglected employment sectors - 
as shown in the mismatch between the proportions of the working population in 
retail and logistics, compared with funded places - or the constant complaints of 
insufficient flexibility in training delivery. The LSC might take a more assertive role 
here, working to ensure that vocational provision genuinely addresses the needs of 
employers. We met some concerns that the growth of vocational options in schools 
might fall short of industry standards. 

4.16 The difference in the social and educational mix of local communities suggests that 
there should be scope to tune performance indicators to local conditions. A number 
of respondents commented that local communities did not seem to have an input 
into the targets for their area. However, one large northern college did help develop 
local targets and goals - for NEETs and HE entry, for example. This may be a 
consequence of their participation in a Local Strategic Partnership that took in a 
wide variety of players, not just in the education world. In our view, it also reflected 
their position as a semi-monopoly provider in the district. We consistently found 
that large colleges were able to take a broader view of local need than smaller 
institutions. They knew the job was theirs because it could be no-one else’s. In the 
absence of a dominant supplier, someone needs to allocate the jobs to the local 
network. 
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5 . 



Recommendations 



5.1 We suggest that there are two suites of indicators for the sector. The first should 
maintain the current focus on provider performance. Those currently set - student 
outcomes and satisfaction, recruitment volumes, employer engagement - and their 
planned successors in the new LSC performance measures seem appropriate for 
this task. But there should also be a set of area targets derived from the PSA 
targets and allocated to LSCs and LEAs to deliver. They will require as much work 
as is going in to the development of the institutional performance indicators at the 
moment, but there are some obvious starting points: 

■ participation in education 16-19 and 19+ 

■ attainment of qualifications by age (linked to PSA), level and sector 

■ participation and attainment of specific groups 

■ appropriateness of level of provision to local needs 

■ match of curriculum areas to local labour market. 

5.2 Deriving these targets will require better information than we currently have. What 
is needed is for every LSC area to build the targeting on the figures that show 
overall attainment, both at 19 and for adults, across its population. We hope that 
the admin database mentioned above can help in this task, providing better 
information than the sample interview processes currently used. Its advocates 
claim it can be cut three ways - by home address, location of pre-16 provider, and 
location of post-16 provider - to provide effective analysis of issues and 
performance. Better information is needed on progression, and targets set for 
schools to judge how well they encourage their youngsters to stay in education till 
18 - and not just in their own sixth form. A good school will encourage lifelong 
learning for all its young people. This is a proposal in the government’s current 
plans for schools. Similarly, colleges need much better data on the employability of 
their students 18 . Better information on destinations could become part of Foster’s 
published information set. This is not just a matter of the technical capabilities of 
Ml systems, though this has rightly exercised the FE Review. We heard from some 
areas that some local providers were reluctant, on marketing grounds, to share 
their data on their provision and success. This is simply indefensible, and we 
suggest that information sharing becomes a condition of public funding. 

5.3 In a number of areas, information and aspirational targets exist, but we did not find 
that they impacted on providers at all. After an area has established its current 
position, and set its targets, it needs to be followed with a delivery plan that 
recognises the distinctive roles of local providers, and would give some stiffening to 
the StAR and JAR process. We see this as a central job for the LSC delivery plan, 
which needs to include public statements about the roles of various providers, and 
the effect this has had in the allocations planning process. Some providers pointed 
out that this document comes out after institutions’ strategic plans have been 
completed: this needs to be reconsidered as part of the LSC business plan cycle. 
The new annual review process would provide the obvious place for the dialogue 
about an institution’s role in achieving the area goals to take place. 

5.4 We did find some effective collaborative working, but in many areas the current 
arrangements fail to deliver the necessary results. We need to address this strategy 
vacuum. A lead partner in the locality needs to be empowered to identify the mix of 
provision that is needed and make the adjustments needed to ensure a proper mix 
of provision. Given its purchasing power, the most obvious agency is the LSC. 



18 The NAO report suggests that LSCs could have a role in sponsoring joint research with its providers. This seems a 
good place to start. 
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Travel-to-study patterns may mean that the LSC should work via the local area 
teams now being assembled, or in consortium with another LSC where boundary 
issues are important. Level 2 work especially needs to be incentivised - by firmer 
planning or better cash allocations. Changes are needed, too, in employment- 
related work: either moves into new areas, or changes in the style of delivery. It 
may therefore be helpful to divide the analysis into 14-19 and 19+. However 
organised, though, the LSC’s approach must be clearly integrated into their overall 
business plan, with clear targets, power enough to drive action - whether funding 
or allocation incentives - and be time-limited. 

5.5 The method of making the changes needs thought. What we emphatically do not 
need is a heavy-handed planning intervention buying so many places on particular 
programmes. But mechanisms currently exist that are being under-exploited - for 
example, the provider review process. Providers complain that - though 
enormously improved since its early days - provider review does not impact on 
their strategies. It seems to be solely a review holding colleges and other agencies 
to account for what happened in the past, dislocated from allocation decisions. As 
with performance indicators, local LSC officers seem more concerned with a narrow 
view of quality - what was the success rate on a particular programme - than 
getting the local balance right, a task for which they are surely better placed. LSCs 
do not analyse the plans of different providers and feed back the information in 
order to avoid overlap or gaps. Bringing a view on local need to the annual review 
discussions would go some way to reconciling the individual corporate nature of 
schools and colleges with the desire for collaboration and partnership. It would also 
help fulfil the ambitions of the Foster Report when it asks (paras 184-189) for the 
LSC to ‘improve coherence and integration at a local level’ and ‘involve providers in 
their assessment of what is needed and should be provided’. We echo Foster’s 
desire that the approach should be ‘light touch’, but we found in most areas there 
was (especially with schools) currently no touch at all. 

5.6 So the main engine will be planning. But it is not the only one. The new buzzword 
‘contestability’ may have virtues here. Recently the LSC announced the protocol for 
choosing between providers competing to open new post 16 provision. This could 
serve as another model for this process, but regrettably the current consultation 
draft focuses on Level 3 work, where there is little evidence of need. We suggest 
the model should be used to develop lower level 16-19 provision, and capacity for 
job-related training. Another interesting method might be found in the idea of 
‘pooled funding’ mentioned in the 2006/7 Grant Letter (DfES 2005) and piloted in 
the collegiates of Birmingham and Solihull LSC, though we recognise that the 
reduction in the size of discretionary funds makes this less powerful than it might 
have been. 

5.7 Whichever mechanism is chosen, it must be driven through with purpose. Failure to 
make the needed advances should be regarded as a rebuke to the LSC, and might 
lead to re-contracting the task to a new consortium or provider. Local LSCs cannot 
be the only part of the learning and skills system where poor performance does not 
involve decisive corrective action. 

5.8 We were encouraged by the increasing use of value-added indicators in judging 
success in delivering the 16-19 full-time academic curriculum. We found no 
evidence of their use elsewhere. This implies that we must find another way of 
making secure judgements of the performance of institutions and departments 
working in different environments and with different student intakes. We suggest 
the further development of statistical families - what one of our respondents called 
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institutions ‘with the same DNA’. The LSC currently has separate benchmarks for 
deprived and non-deprived colleges, and we feel these can be developed further. 

5.9 We noted above the comparatively underdeveloped state of the use of 
performance indicators inside institutions, and the unsophisticated interpretations 
used by managers. As this is to remain a key part of our sector management for the 
foreseeable future, we would argue the following: 

■ College and school management training should include a module on 
performance management that looks at the strengths and weaknesses of 
the current approaches, and encourages the future leaders of the sector to 
use them judiciously to improve performance - internally and across their 
community. 

■ LSC officers too should be made aware of the strengths and limitations of 
the numbers they use. 

5.10 Our respondents were aware that schools and FE measured success rates 
differently. There seems no reason for this, and we join the chorus of people asking 
for a common approach, without fudges about ‘coherence’ or ‘comparability’. Until 
we do this, it will not be possible to make sensible judgements either between 
providers or to meet community need. 

5.11 We would support the ability and capacity of local groups - whether LSCs or 
those with whom they contract - to supplement national priorities with indicators to 
represent their own community needs. It is plain that (for example) NEETs or HE entry 
are bigger problems in some areas than others. This would mean two things: 

■ work on delegating the PSA targets to local conditions 

■ additional targets and measures to address specifically local issues. 

5.12 Having worked for several months with a substantial sample of post-16 providers 
we are convinced that the work of raising quality within individual providers is in 
good hands. Managers are driving the standards agenda with determination. This 
perception is supported by the figures that indicate substantial progress improving 
success rates in colleges, work-based providers, schools and adult education 
services. Ironically, it is in the area of quality and volumes that the local LSC is most 
assertive with its providers; it is least involved in the area of curriculum mix and 
levels: it is here where more work is needed in assessing and meeting local needs 
as a whole. We would look to local LSCs - or the selected delivery consortium - 
piloting our approaches, setting targets and indicators for their areas and driving 
them through their planning processes. They are already working more assertively 
as part of the ‘plan-led funding’ approach in ways that would help. Such initiatives 
should be observed and evaluated nationally to assess their success in meeting 
the needs of the locality and the nation. Few, if any, of our recommendations to 
remedy the position require new legislation. 

5.13 Since the conclusion of our field work, two substantial documents have been 
published relating to 14-19 education. The Nuffield Review of 14-19 Education 
and Training saw the system as ‘divided, competitive and (only) weakly 
collaborative - features which appear to depress participation rates because they 
do not tackle 16-plus selection’. It argues for a ‘strongly collaborative 14-19 
learning system... conceived and delivered locally and sub-regionally... (with) a well- 
defined sense of the local area...’ (Hayward 2005). Our proposals would look like 
this ‘strongly collaborative’ model, and provide a fresh way of reconciling the 
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individualism of schools and colleges with the collaboration and common purpose 
sought by policy-makers. The Foster Review (Foster 2005) touched a number of our 
points - not just the need for radically improved information, but also the need for 
a real strategy for the delivery system. Its executive summary points out the 
following: 



■ that ‘incorporation liberated individual colleges but failed to provide a 
basis for locality strategy’ 

■ the need for ‘clarity at a local level... served by a clear and separate 
commissioning role built around local LSCs’ 

■ ‘the funding methodology should draw on the national learning 
requirement and its local components ensuring that learning activity 
is rewarded for impacting on those requirements’ 

■ ‘an improved inspection methodology (is needed that) that focuses on 
learner experience, value-for-money, and (our emphasis) the 
coherence of the locality's learning offer and impact. ’ 

5.14 We hope that our argument - that we need to retune our performance measures to 
judge how well our young people and adults are doing, rather than how well our 
schools and colleges are doing - is an idea whose time has come. It is not an easy 
path. It needs a sharper look at strong collaboration, backed by funding and 
planning and uncluttered by unhelpful ideas of autonomy. It requires us to look 
outside the current indicators to create new measures of success, and to use the 
planning powers to achieve a better balance of provision. We need, as well, to 
retune our ideas of educational worth and prestige. To make these changes will 
require the fresh look at performance indicators and local delivery systems that we 
recommend. But the prize - a better qualified, more efficient and more inclusive 
society - is worth the fight. 
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Appendix 2 - methodology and survey questionnaires 

A2.1 The research took place between May and October 2005, with the bulk of the work 
undertaken by Adrian Perry and Matthew Simpson. Adrian Perry is Visiting Senior 
Fellow at the University of Sheffield, and has worked for a number of clients - 
including the DfES and LSC - since retiring as an FE principal. Matthew Simpson 
has worked for many years in education, both in colleges and LEAs, and latterly as a 
Chief Education Officer. Fie undertakes consultancy commissions both in this 
country and overseas. 

A2.2 At a national level, we spoke to policy staff in the LSC national office, the DfES, 

Ofsted and the Cabinet Office, and supplemented them with interviews with senior 
staff in the Secondary Pleads Association, NATFPIE, Ofsted and the National 
Institute for Adult and Continuing Education. To get the picture on the ground, we 
chose four LSC/LEA areas: Nottinghamshire, Birmingham and Solihull, Central 
London and South Yorkshire. In each of these areas we spoke to LSC staff, LEA 
officers and a representative sample of senior staff - usually at assistant principal 
or head of sixth form level - from a number of schools, sixth form and FE colleges. 

A2.3 We also talked to a range of other partners - two Connexions managers, a learning 
partnership, the Association of Colleges and a sixth form consortium. To check our 
findings, we later interviewed a range of middle managers from other regions, and 
convened a small focus group to consider the draft report. We also talked to some 
LSCs outside our sample - Surrey and Hampshire/Isle of Wight - who had made 
contributions to the issues we were looking at, and cross-checked our findings with 
colleagues from York Consulting, who are undertaking work for the DfES on the 
characteristics of improving colleges. The Responsive College Unit was generous 
with comment, and drew our attention to evidence coming from some of its 
contracted work. We are deeply grateful to our provider contacts for their time and 
insights. We hope we have represented their views correctly - all interview 
transcripts were checked back with respondents - but, as ever, the responsibility 
for errors and omissions is ours alone. They are listed here: 

Barking College 

Barnsley College 

Bilborough College 

Broxtowe College 

City and Islington College 

City College Coventry 

City College Norwich 

King Edward VII School Sheffield 

LASWAP Sixth Form consortium 

Leicester College 

Longley Park Sixth Form College 

Newham Sixth Form College 

North Nottinghamshire College 

Orpington College 

Sheffield College 

Solihull College 

Solihull Sixth Form College 

South Birmingham College 

Truro College 

West Nottinghamshire College 
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A2.4 The method involved structured interviews with all our respondents. The early 

interviews were conducted face-to-face by both researchers to establish a common 
approach and check that the questionnaire was right. The bulk of later work was by 
telephone interviews around the questionnaires. We attach the survey instruments 
used in Appendix 4. A draft was prepared on the basis of our interviews, and 
discussed with a group of critical friends in mid-October before the preparation of 
the final report. 
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Appendix 3 - the problems with performance indicators 



A3.1 In para 2.4 of our main report, we gave a brief resume of the drawbacks of 

performance indicators: it was decided to leave a more extended treatment of the 
issues to a separate section in order to maintain the shape of the report. The 
general reservations about indicators follow. 

A3. 2 Social factors: The ability to achieve strong outcomes often depends on social 
context. Police forces in a tough inner-city environment will struggle to match the 
low crime-rates of the suburbs, for example. There is a strong relationship between 
school outcomes and indices of deprivation - often measured as the proportion of 
pupils entitled to free school meals. This would make unadjusted comparisons 
between differing areas deceptive, and so there are various ways to make 
adjustments. One is to look at how the performance of providers - schools, police 
forces, LSCs - compare with those in similar circumstances - with ‘the same DNA’, 
as one of our respondents put it. For example, separate LSC benchmarks for 
student success exist for disadvantaged colleges. An alternative approach is to 
create a numerical measure of deprivation that can be used to adjust the raw score 
of a provider or region. However, journalists and politicians like unadjusted league 
tables - Kenneth Clarke when a minister famously preferred his statistics ‘raw 
rather than cooked’. 

A3. 2 Creaming: There is a danger that some agencies, knowing that they will be judged 
on a numerical outcome, will choose the easier tasks to do. For example, surgeons 
may be reluctant to operate on critically ill patients; in education, agencies may 
avoid work with disengaged or less able students. This is sometimes known as 
‘creaming’. The value-added techniques discussed in our text will have the effect of 
discounting the benefits of creaming: if a college recruits only able students, it will 
be expected to get outstanding results. 

A3. 3 Measurement: You have to choose something to measure to assess progress in 
health care, education, or crime prevention: performance indicators can be 
sensitive to the particular numbers chosen. Data can prove difficult to collect, or 
can be deceptive. There are a number of problems in the learning and skills sector 
in this respect. Few would dissent from the use of qualifications as the proxy for 
educational success, but it does create distortions. Some learners are counted as 
failures when they leave to take up employment before the exam. Success rates 
are sensitive to course design. For example, recorded student retention in the 
sector rose when the two-year A-level programme was divided into two single-year 
chunks: students who passed AS and then left were considered as successes, 
whereas they would have counted as drop-outs the previous year. There are also 
issues in the sector around measuring employer engagement, around what is an 
ESOL success, and around what is meant by a Level 2 attainment. 19 

A3. 4 Precision and uncertainty: There is the additional problem that numbers are rarely 
precise. International comparisons are particularly fragile, with Peter Robinson of 
the Institute for Public Policy Research (IPPR) advising of an accuracy range of +/- 
5%. An example from Simon Briscoe’s ‘Britain in Numbers’ illustrates the issue 
within the UK: if you rank our universities by the various quality measures, there is a 
range of plus and minus uncertainty around a given score. This range is in fact 
sufficiently large that there is a chance - not a big one, but a chance - that the 9 th 
ranked university is worse than the 110 th ranked one (Briscoe 2005). Within 



19 Changing the measure of Level 2 attainment from the Labour Force Survey to the administrative database (see para 
4.13) will depress the Level 2 figure by 5%. If we say that the Level 2 attainment must include English and Maths - which 
is an entirely reasonable view - the figures will fall further. 
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institutions, as sample sizes get smaller, figures need to be assessed with care. In 
a class of 20, one student result moves the stats by 5 percentage points. This is the 
issue of statistical significance. A number of our respondents were aware of this 
problem, but careful that it did not give excuses for poor performance as a ‘glitch 
year’. 

A3. 5 Gaming: Managers judged on numerical indices can take action which makes the 
numbers look good without improving the public service. There have been 
examples of false reporting - GPs exaggerating their patient list to maximise 
funding, police forces re-categorising crimes - and of cheating, with schools 
coaching pupils through key stage tests. But this is rare: more common is what is 
known as ‘gaming’ - activity where managers consciously modify policies and 
procedures to hit the targets while leaving underlying quality unchanged, or even 
worse. Train companies stretch journey times to avoid penalties for late arrival. GPs 
asked to achieve appointments within 48 hours refuse to take advance bookings. 

In the Soviet economy, glass factories which had been set a tonnage target 
proverbially produced thick glass, but moved to thin glass when asked for square 
metres. 

A3. 6 Distortions: A more common problem comes when those tasked with hitting targets 
could concentrate on them to the exclusion of other worthwhile activity. This does 
not involve dishonesty - managers are keen to secure what they feel their masters 
want - and it is different from gaming, because the measured aspect of the service 
is certainly getting better. But it can lead to what are known as ‘perverse 
outcomes’. It is sometimes claimed, for example, that hospital administrators 
schedule unimportant but long-standing operations rather than urgent immediate 
ones to stay within permitted waiting times. Some schools react to being measured 
on the proportion of students gaining 5 A-C GCSE passes by working intensively 
with those on the margin of that achievement, ignoring the very bright and the less 
able. Closer to home, the increased emphasis placed on FE success rates and 
retention of the mid-90s was accompanied by a sharp increase in the number of 
one-day courses. It is also alleged that the choice of progression to apprenticeships 
as the badge of success of E2E programmes has led to pregnant youngsters and 
those awaiting court sentence being excluded from a programme that might help 
them back on the rails. 

A3. 7 Lastly, cost: The collection of figures takes time and money that could be out of 
proportion to the benefit. Colleges often complain about the number of 
management information staff needed, and their academic staff resent time spent 
away from students and teaching. The complaint is made more forcefully when it is 
felt the information is of poor quality (student destinations in general further 
education, for example) or is not relevant to the recipient (for example, the staff 
qualifications database sent off to LSCs). 



31 




Appendix 4: the questionnaires used in the study 



Project SR 723 

Area Pis: ‘Pis and Performance’ learning 

(Performance Intelligence Strategic Survey) and skills 

development 

agency 



Questionnaire for providers 



Organisation: 

Colleague: ... Position:... 

Researcher: Adrian Perry/Matthew Simpson Date: 



Approx FTE students: Approx annual expend: £ M 

Approx balance of Level 2/Level 3/Level 4 work: / / .... 



General 


1 


What are the key Pis in the college/school? Why these? 




a) Who in the institution would be described/recognised as ‘leading’ on Pis? 

b) Who determines PI policy? 

c) What handle do governors have on performance indicators? (Prompt: Which Pis 
are reported to the governors? What is their attitude? Have they established internal 
Pis for specific result areas?) 


2 


Which areas of performance are currently seen as priorities within the institution? 
Why? (External v internal? Funding imperatives v Quality/inspection?) 


3 


Are the Pis used as indicators or as targets? 


4 


Can you give any examples of management decisions prompted by Pis? Do they 
make you do anything you would not otherwise do? 




Take an example. There has been an increasing emphasis on college success rates 
and retention - how have you reacted? 




Were the steps you took beneficial in meeting need or cosmetic to make the 
numbers better? Or a bit of both? 
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Have you noticed any action within departments that might be regarded as 
‘massaging the figures’ rather than improving the service for students? 


5 


How important are ‘league tables’ to the college/school? To staff? To customers - 
employers and students? 


6 


What’s your general view of performance indicators in the sector? Generally, does 
their existence add or subtract value? Any particular difficulties of fairness, 
resourcing or timing? 


External Pis 


7 


What are the most recent inspection grades of the college/provider? 


8 


What are the most important elements in an (Ofsted & ALI /Ofsted) Inspection of the 
institution? Which are the ones that the institution’s behaviour might be influenced 
by? 


9 


Has there been an area inspection here? Of the elements in an Ofsted Area 
Inspection, are there any of which you are particularly mindful - which the 
institution’s behaviour might be influenced by? 


10 


What external Pis do you have to return to your sponsors (LSC, LEA, ESF, anyone 
else)? 




Are there any of these external Pis - that is, demanded by your funders or 
inspectors - which are particularly difficult for your institution: is the problem 
- ease of collection? - relevance? - fairness? 

Which particular returns does this refer to? 


11 


Are there any data that you think should be collected that currently are not? 
(prompt - destination data, fee income, responsiveness, diversity/social 
engagement) 


Internally Determined Pis 


12 


Are there any additional Pis which the institution uses and looks at for its own 
purposes? (Prompt: departmental level? Ben Johnson Hill benchmarking?) 


13 


If so, are there are tensions between those and the externally-required ones? 


National PSA targets 


14 


There is a range of national targets set by the government to the DfES. To what 
extent are senior staff and the governors generally aware of these? 


15 


How are these aims expressed? 




Do there seem any tensions between those national aims and the directions in 
which specific Pis drive the institution? 
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Area-Based Intelligence 


16 


What do you see as your area (catchment)? 

What in your view are the major educational/skills issues in this area (prompt - 
supply-side issues, participation of young people/NEETS, attainment of young 
people, unemployment, adult skills, industry specific skills)? 

Who else expresses a view on these issues? (LSC? RDA? LEA/EDP? Employers? 
Connexions? Learning Partnerships?) 

Have you (or anyone else) attached numbers to these problems? 


17 


Is there any experience of monitoring and/or target-setting at a local trans- 
institutional level, eg at the level of Local Learning Partnership, or LEA? 


18 


If so, has it had any tangible outcomes for this institution? 


19 


Would you see merit in (further) development of area-based PI - assuming any 
institutional energy required for that would be offset by reducing current Pis? 

Is there a partnership that could energise all the providers to jointly achieve shared 
goals? 


Concluding 


20 


What is being sought is sometimes referred to as a ‘line of sight’ from national 
targets, through area and regional priorities, to institutional priorities. 

How could that best be achieved? 

(Prompt: incentivisation through resourcing/Foster on governance & accountability, 
etc) 


21 


Do you have any other comments or questions? 



Survey Instrument for MIDDLE MANAGERS 

Organisation: 

Colleague: ... Position: ... 

Researcher: Adrian Perry/Matthew Simpson Date: 

Stress confidential nature of answers 

1. What are the key performance targets your institution has to achieve? 

2. Why do you think these are important? 

3. How well are you doing on these measures as an institution, do you think? 

4. Does your section/team have any targets to hit? 
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• volumes/recruitment 

• success/retention/pass-rates 

• others (employer engagement/diversity/commercial earnings?) 

5. How are your targets/ perform a nee measures derived? From last year’s attainment? 
From college-wide targets divided up? Are they agreed with you? 

6. What happens if you perform well or poorly in relation to these targets? 

7. What factors outside your control affect the figures? 

8. Performance numbers can be improved in various ways - improved 
teaching/changing staff, setting targets for improvement, improved student 
support, changing to more appropriate courses and assessment, moving to more 
short courses, better admissions guidance, stricter exam entry, counting the figures 
differently, better equipment or marketing. Can you give an example of how you 
have improved performance in your section - which of these strategies (or any 
others) did you use? 

9. Do you delegate performance targets to sections in your team? Do you worry that 
they might take actions just to improve the numbers, rather than improve the 
service for students/the community? Have you seen this happening ether in your 
current institution, in a previous one, or a competitor? 

10. What’s your overall view of the growing use of targets and indicators to manage and 
assess performance? Is it a helpful tool or a misleading technique? 



Survey Instrument for SPONSORS (LSC, LEA) 



Organisation: 

Colleague: Position: 

Researcher: Adrian Perry/ Matthew Simpson. Date: 2005 



Performance indicators - the basics 

1. What performance indicators do your require of your providers? 

2. Why are these chosen? 

4. Do you have any local discretion about the suite of Pis used? 

5. What do you feel about that? Are there local features that are missed by the 
national suite of Pis? 

6. What’s your general view of performance indicators in the sector? Generally, does 
their existence add or subtract value? Are there issues about (eg) timing, accuracy, 
fairness, relevance? 

Using performance indicators 
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7. What do you do with the information? 

8. How important are Pis in the assessment of provider performance (as against eg 
development plans, inspection reports)? 

9. Is it your view that providers act differently because of what you require of them? In 
what way? 

10. Have you any fast improvers locally? How do you think they have achieved their 

improvement? Evidence of pedagogic improvement? Of data 

massage/gaming/creaming? 

Analysing local needs 

11. What do you see as the main education and training issues in this area? 

12. Why do you pick these issues out? Were they identified in any StARs or Area-Wide 
Inspections? 

13. Is it area performance or institutional performance that is the major concern? 

14. Are your key issues represented in your local plan? Are they represented in Pis or 
targets set to your providers? Talk us through how you hope that your analysis will 
affect the actions of providers. 

Meeting local needs 

15. Do you feel that if providers hit their volume and quality targets, the needs of the 
community will be well met? Or could you envisage a situation where all local 
providers were - in their own terms - successful but some of the needs of the area 
were being neglected? How could you get around that? 

16. We are trying to see whether the Pis that are demanded of providers might cause 
them to act in ways that work against the best interests of the area - for example, 
avoiding excluded youngsters because they might lower retention rates. Is this a 
worry here? Is there any evidence that it is a problem? 

17. What sort of indicators or targets would support improvements locally? 

18. If you established area-based indicators (say, for Level 2 attainment at 19), how 
would they be delivered? Are there any partnerships, for example, that might take 
ownership? 

PSA targets 

19. To what extent is the local LSC, and local providers, aware of PSA targets? Do they 
influence local decisions and behaviour? 

20. What is being sought is sometimes referred to as a ‘line of sight’ from national 
targets, through area and regional priorities, to institutional priorities. How could 
that best be achieved? 

21. Are there any other comments you’d like to make about the usefulness of 
performance indicators? 
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22. Thanks - what happens next 



Survey Instrument for PARTNERS (Learning Partnerships, Connexions, etc) 
Performance indicators - the basics 

1. What performance information is available to you about local ed & training? 

2. Do you have any influence on the range of Pis used? 

3. How useful is that info to you? 

4. Are there local features that are missed by the national suite of Pis? 

5. What’s your general view of performance indicators in the sector? Pis have 
sometimes been described as an energy-trap or as a self-justifying industry. 
Generally, does their existence add or subtract value? Are there issues about (eg) 
timing, accuracy, fairness, relevance? 

6. Is it your view that providers act differently because of the information that is 
required of them? In what way? Evidence? 

Analysing local needs 

7. What do you see as the main education and training issues in this area? (eg overall 
success rates? Success rates in specific curriculum areas/levels/type of learner (by 
age, ethnicity, gender)? Recruitment/participation? Employer engagement? 
Financial performance of providers? Others?) 

8. Why do you pick these issues out? Were they identified in any StARs or Area-Wide 
Inspections? (Copies available?) 

9. Is it area performance or institutional performance that is the major concern? 

10. Do you feel that if providers hit their volume and quality targets, the needs of the 
community will be well met? Or could you envisage a situation where all local 
providers were - in their own terms - successful but some of the needs of the area 
were being neglected? How could you get around that? 

11. Are your key issues represented in local plans? Are they represented in Pis or 
targets set to your providers? 

12. Talk us through how you hope that your analysis will affect the actions of providers. 

13. We are trying to see whether the Pis that are demanded of providers might cause 
them to act in ways that work against the best interests of the area - for example, 
avoiding excluded youngsters because they might lower retention rates. Is this a 
worry here? Is there any evidence that it is a problem? 

14. What sort of indicators or targets would support improvements locally? 
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15. If you established area-based indicators (say, for Level 2 attainment at 19), how 
would they be delivered? Are there any partnerships, for example, that might take 
ownership? 

PSA targets 

16. To what extent is the sector locally aware of PSA targets? Do they influence local 
decisions and behaviour? 

17. What is being sought is sometimes referred to as a ‘line of sight’ from national 
targets, through area and regional priorities, to institutional priorities. How could 
that best be achieved? (Prompt: incentivisation through resourcing/Foster on 
governance & accountability, etc) 

18. Are there any other comments you’d like to make about performance information & 
intelligence? 
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delivering quality and choice 

how performance indicators help and how performance indicators hinder 



For all learners, but for those aged 14-19 in particular, 
collaboration between schools and colleges is increasingly 
seen as the only way to provide effective choice; but at 
the same time the key performance indicators focus on 
individual institutions, not the offer in an area. This report 
explores the tensions between a partnership agenda and 
the current emphasis on institutional performance. 

If this report is relevant to the work you do, you may 
also be interested in: 

Modelling Progress toward the Level 2 Target 

Increasing the number of adults qualified to at least Level 2 
is a key target for the DfES and LSC. A study for LSDA by 
Frontier Economics shows that we are set to miss the target, 
and, by analysing who currently lacks a Level 2, suggests why. 

A forthcoming study by NFER for LSDA 

(LSRC 693) 

This will examine whether the pattern of post-16 provision 
in an area affects the participation and achievement of 
the whole post-16 cohort. Does the presence of school 
sixth forms improve participation? Do selective systems 
work better than tertiary solutions? 




